Image Processing Method, Image Recording Method, Image Processing Device and Image File Format

ABSTRACT

A recording process records low-resolution video data while obtaining a high-resolution image in a window region being a portion of an entire image. Then, the process learns, as a resolution conversion rule, a resolution increasing parameter by using the high-resolution image. In a resolution increasing process, the resolution of the recorded low-resolution video data is increased by using the resolution increasing parameter learned in the recording process.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.11/789,010 filed on Apr. 23, 2007 which is a continuation ofInternational Application PCT/JP2006/315655, filed on Aug. 8, 2006. ThisNon-provisional application claims priority under 35 U.S.C. §119(a) onPatent Application No. 2005-262610 filed in Japan on Sep. 9, 2005, theentire contents of which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing technique, and moreparticularly to a technique for resolution conversion such as increasingthe resolution of video data.

2. Description of the Background Art

With the spread of digital image devices such as digital video camerasand digital still cameras, it has become common that high-resolutiondigital images are handled by various types of input/output devices.Particularly, with still images, the image pick-up device of an ordinaryconsumer digital still camera has five megapixels or more, and someproducts include an image pick-up device having over ten megapixels. Itcan be said that sufficiently high resolutions have been realized fordigital photography applications.

As the demands for higher resolutions for still images have been quitesatisfied, it is now expected in the field of digital image applicationsthat there will be more demands for higher resolutions for video data,particularly, for video data such as movies where each frame image is ofsuch a high resolution that it can be used as a good-quality stillimage. There are two fields of application to which image resolutionincreasing techniques are applied, i.e., image input systems such ascameras and image display systems such as TV sets. The present inventionis directed primarily to image input systems.

Increasing the resolution of video data in an image input systeminvolves the following problem. Where video data of an HD (HighDefinition) TV or better quality is desired, it is very difficult toread out all pixels at a video data frame rate of about 30 fps due tothe large number of pixels even if the image pick-up device has enoughpixels. If such a high-speed read-out process is performed forcibly, theequipment will consume excessive power and generate excessive heat.Therefore, with state-of-the-art techniques, it is difficult to recordvideo data with a high resolution every frame, and it is possible onlyto obtain a high-resolution image per a few frames of video data.Researches have been made for the use of various image processingtechniques after images are recorded.

More specifically, a conventional technique in the subject field is forobtaining video data of a high resolution in both the time and spacedomains from video data that has a high resolution in the time domainbut a low resolution in the space domain, by using images that have alow resolution in the time domain but a high resolution in the spacedomain.

Patent Document 1 (Japanese Patent No. 3240339) discloses a techniquefor producing high-resolution video data based on low-resolution videodata and high-resolution still images that are being recorded. With thetechnique of Patent Document 1, high-resolution still images areassociated with samples of frames of low-resolution video data with apredetermined sampling interval therebetween so as to spatiallycompensate for the low-resolution video data to thereby increase theresolution of the video data.

Patent Document 2 (Japanese National Phase PCT Laid-Open Publication No.2005-522108) discloses a technique as follows. A scene is recorded toproduce low-quality image data while a portion of the scene is recordedto produce high-quality image data. Then, the high-quality image dataand the low-quality image data are used as a learning pair in a learningalgorithm to thereby determine quality-improving function parameters,based on which the high-quality image of the rest of the scene isderived.

However, the conventional techniques have the following problems.

The technique disclosed in Patent Document 1 is to produce video data ofa high resolution in the space domain based on video data of a lowresolution. Specifically, video data and still images are associatedwith each other at discrete points along the time axis, and thereforeinformation for frames with which still images have already beenassociated is used for video data frames of which there is noassociation information. Then, similar signal level edges are searchedfor, which if found are considered to indicate translation-like movementof an object. Then, a motion vector searching process is used todetermine the pixels to be compensated for in the space domain. Theproblem is that the searching process imposes a heavy load, and it mayresult in incorrect pixel associations. If the object deforms or turnssimultaneously with its movement, the process may not find points to beassociated with each other, whereby the process fails.

With the technique of Patent Document 1, the high-resolution image andthe low-resolution image are read out at the same speed. If theresolution conversion factor between the video data and the still imagesis up to about two (i.e., two horizontally and two vertically), theprocess of reading out high-resolution images does not take excessiveamounts of time. However, if the conversion factor is increased to aboutfour, the total area of a high-resolution image to be read out is 16times as great as that of a low-resolution image, and the process ofreading out high-resolution images will take excessive amounts of time.As a result, there will be a significant increase in the number offrames to be dropped from the recorded video data, and the quality islikely to deteriorate due to the frame dropping in the video data.

With the technique of Patent Document 2, the position where thehigh-quality image data is recorded is fixed, for example, substantiallyat the center of the scene. Therefore, the quality-improving functionparameters are determined based on the image characteristics at thefixed position. Thus, high-quality images may not be derivedappropriately if the image characteristics at the fixed position aredifferent from those of other positions. This presents a significantproblem especially when increasing the resolution of video data, wherebyit is likely that a sufficient precision will not be obtained in theresolution conversion.

SUMMARY OF THE INVENTION

It is an object of the present invention to realize an image process forincreasing the resolution of video data, wherein a high-precisionresolution conversion is realized even with a high resolution conversionfactor of about four or more, for example.

According to the present invention, in a recording process of recordinglow-resolution video data, a high-resolution image is obtained everyframe in the window region being a portion of the entire image. Then,the process learns a resolution conversion rule by using thehigh-resolution image in the window region. The position of the windowregion is changed every frame. In the resolution increasing process, theresolution of the recorded low-resolution video data is increased byusing the resolution conversion rule learned in the recording process.

According to the present invention, a high-resolution image is obtainedonly for the window region being a portion of the entire image, wherebythe pixel read-out process does not require a long period of time.Therefore, it is possible to record the low-resolution video datawithout dropping frames. Since the high-resolution image is obtained foreach frame, the high-resolution image is not shifted in time withrespect to the low-resolution image, thus allowing for the process toappropriately learn the resolution conversion rule. Since the positionof the window region changes every frame, the resolution conversion ruleis learned based on data from a large area of the image, but not datafrom a limited area of the image. Thus, the process can appropriatelylearn the resolution conversion rule. Moreover, the high-resolutionimage does not need to be stored, and only the low-resolution video dataand the resolution conversion rule need to be stored, whereby it ispossible to considerably reduce the amount of information to be storedas compared with conventional techniques. In the resolution increasingprocess, the resolution conversion rule, which is learned for the windowregion being a portion of the entire image, is used for the entire imageof the recorded low-resolution video data. Unless the imagecharacteristics of the object in the window region are significantlydifferent from those outside the window region, it is possible torealize a high-precision resolution conversion across the entire image.

In the present invention, the process may produce the low-resolutionimage in the window region through an image filtering process such assub sampling or averaging on the high-resolution image. Then, it is notnecessary to read out the low-resolution image for the window region,and it is therefore possible to further reduce the total number ofpixels that need to be read out.

In the present invention, the window region may be moved every frame sothat the entire image is scanned by the window region over a pluralityof frames. Then, the process learns the resolution conversion rule forthe entire image over a plurality of frames, thus realizing theresolution increasing process with an even higher precision.

Thus, according to the present invention, it is possible toappropriately learn the resolution conversion rule and to realize aresolution conversion with a high precision. Moreover, the pixelread-out process does not take a long period of time, whereby thelow-resolution video data can be recorded without dropping frames.Moreover, since it is not necessary to store the high-resolution image,it is possible to significantly reduce the amount of information to bestored. Therefore, it is possible to realize a high-precision resolutionconversion even with a high resolution conversion factor of about fouror more, for example.

The present invention is not limited to resolution increasing processes,but is applicable to resolution conversions in general. The presentinvention can also be effectively used in producing a low-resolutionimage to be displayed on a low-resolution display system such as amobile telephone, for example.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart showing a recording process in an imageprocessing method according to a first embodiment of the presentinvention.

FIG. 2 is a flow chart showing a resolution increasing process in theimage processing method according to the first embodiment of the presentinvention.

FIG. 3 is a conceptual diagram showing an example of a recording processand a resolution increasing process according to the first embodiment ofthe present invention.

FIG. 4 is a flow chart showing an example of a process in step S13 ofFIG. 1.

FIG. 5 is a flow chart showing an example of a process in step S23 ofFIG. 2.

FIG. 6 shows an exemplary method for conversion to a texture feature instep S132 of FIG. 4.

FIG. 7 shows a converted texture feature.

FIG. 8 shows an exemplary configuration of an image processing deviceaccording to a second embodiment of the present invention.

FIG. 9 is a conceptual diagram showing an exemplary configuration of amulti-pixel-resolution image pick-up section in FIG. 8.

FIGS. 10A to 10D show an exemplary method for reading out pixels by amulti-pixel-resolution image pick-up device.

FIGS. 11A to 11D show an exemplary method for reading out pixels by amulti-pixel-resolution image pick-up device.

FIG. 12 shows an exemplary circuit configuration of amulti-pixel-resolution image pick-up device.

FIG. 13 shows an exemplary circuit configuration of an imaging pixel inthe multi-pixel-resolution image pick-up device of FIG. 12.

FIG. 14 shows an exemplary circuit configuration of a image-storingpixel in the multi-pixel-resolution image pick-up device of FIG. 12.

FIG. 15 shows an exemplary circuit configuration of a skip scanningshift register in the multi-pixel-resolution image pick-up device ofFIG. 12.

FIG. 16 is an operation sequence diagram illustrating the outline of aseries of operations, i.e., the image-recording operation, the transferoperation and the read-out operation, by the multi-pixel-resolutionimage pick-up device shown in FIGS. 12 to 15.

FIG. 17 is a timing diagram showing the operation during the high-speedV transfer period of FIG. 16.

FIG. 18 is a timing diagram showing the operation during the horizontalread-out period of FIG. 16.

FIG. 19 is a timing diagram showing the output of a selector of FIG. 12.

FIG. 20 conceptually shows a texture conversion process according to thesecond embodiment of the present invention.

FIG. 21 shows a method for producing an analysis code book and areproduction code book.

FIG. 22 shows a configuration of an image display section according tothe second embodiment of the present invention.

FIG. 23 shows a configuration of an image processing device according toa third embodiment of the present invention.

FIG. 24 shows an image format of multi-pixel-resolution compressed videodata according to the third embodiment of the present invention.

FIG. 25 shows a configuration of an image display section according tothe third embodiment of the present invention.

FIG. 26 shows an exemplary configuration of an image processing deviceaccording to a fourth embodiment of the present invention.

FIG. 27 shows a configuration of a video processing server according tothe fourth embodiment of the present invention.

FIG. 28 shows a configuration of an image processing device according toa fifth embodiment of the present invention.

FIGS. 29A to 29D show a process of detecting an object candidate regionaccording to the fifth embodiment of the present invention.

FIG. 30 shows an example of how to set a window region according to thefifth embodiment of the present invention.

FIG. 31 shows a configuration of an image processing device according toa sixth embodiment of the present invention.

FIG. 32 shows a configuration of a multi-pixel-resolution image pick-upsection with a polarizing filter shown in FIG. 31.

FIG. 33 shows a configuration of an image display section according tothe sixth embodiment of the present invention.

FIG. 34 shows an effect of a resolution increasing process according tothe sixth embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A first aspect of the present invention is directed to an imageprocessing method, including: a recording process of recordinglow-resolution video data; and a resolution increasing process ofincreasing a resolution of the low-resolution video data recorded in therecording process, wherein: the recording process includes: a step ofobtaining, in each frame of the low-resolution video data, ahigh-resolution image in a window region being a portion of an entireimage; and a step of learning, in each frame, a resolution conversionrule by using the high-resolution image in the window region; theresolution increasing process includes a step of increasing a resolutionof the low-resolution video data by using the resolution conversion rulelearned in the recording process; and in the recording process, aposition of the window region is changed every frame.

A second aspect of the present invention is directed to the imageprocessing method of the first aspect, wherein: the recording processobtains multi-pixel-resolution video data by taking an image with a highresolution in the window region and an image with a low resolutionoutside the window region; and with the multi-pixel-resolution videodata, the recording process performs an image filtering process on thehigh-resolution image in the window region to produce the low-resolutionvideo data.

A third aspect of the present invention is directed to the imageprocessing method of the second aspect, wherein in the recordingprocess: a motion of an object candidate region is detected in thelow-resolution video data; and a position of the window region is movedaccording to the detected motion of the object candidate region.

A fourth aspect of the present invention is directed to the imageprocessing method of the second aspect, wherein in the recordingprocess: the multi-pixel-resolution video data is obtained while beingsplit into a diffuse reflection component and a specular reflectioncomponent; and the low-resolution video data and the resolutionconversion rule are obtained separately for the diffuse reflectioncomponent and for the specular reflection component.

A fifth aspect of the present invention is directed to the imageprocessing method of the first aspect, wherein the resolution conversionrule describes a correlation between texture feature vectors of imagesof different resolutions.

A sixth aspect of the present invention is directed to the imageprocessing method of the first aspect, wherein the resolution increasingprocess is performed for a subject frame by using the resolutionconversion rule learned in at least one frame preceding or following thesubject frame, in addition to the resolution conversion rule leaned inthe subject frame.

A seventh aspect of the present invention is directed to the imageprocessing method of the first aspect, wherein in the recording process,the window region is moved every frame so that an entire image isscanned by the window region over a plurality of frames.

An eighth aspect of the present invention is directed to the imageprocessing method of the seventh aspect, wherein the window region isone of n regions (n is an integer of two or more) into which an entireimage is divided, and the window region is moved so that the entireimage is covered by the window region over n frames.

A ninth aspect of the present invention is directed to the imageprocessing method of the eighth aspect, wherein the resolutionincreasing process is performed for a subject frame by using theresolution conversion rules learned for n frames including the subjectframe, and by using, for each portion corresponding to the windowregions of the n frames, the resolution conversion rule for the framecorresponding to the portion.

A tenth aspect of the present invention is directed to the imageprocessing method of the first aspect, wherein in the window region, theresolution increasing process synthesizes together an originalhigh-resolution image and an image obtained by the resolution increasingprocess with a predetermined synthesis ratio.

An eleventh aspect of the present invention is directed to an imagerecording method, including a step of obtaining and recordingmulti-pixel-resolution video data by taking an image with a highresolution in a window region being a portion of an entire image whiletaking an image with a low resolution outside the window region, whereina position of the window region is changed every frame in the step.

A twelfth aspect of the present invention is directed to an imageprocessing device, including: a multi-pixel-resolution image pick-upsection for obtaining multi-pixel-resolution video data by taking animage with a high resolution in a window region being a portion of anentire image and whose position changes every frame while taking animage with a low resolution outside the window region; a downwardresolution conversion section for performing an image filtering processon the high-resolution image in the window region to decrease aresolution of the high-resolution image; a low-resolution videorecording section for recording low-resolution video data obtained froman output of the downward resolution conversion section and themulti-pixel-resolution video data; a resolution increasing parameterlearning section for learning a resolution increasing parameter by usingthe high-resolution image in the window region and the output of thedownward resolution conversion section; and a resolution increasingparameter recording section for recording the resolution increasingparameter.

A thirteenth aspect of the present invention is directed to the imageprocessing device of the twelfth aspect, wherein themulti-pixel-resolution image pick-up section obtains themulti-pixel-resolution video data while the multi-pixel-resolution videodata is split into a luminance component and a color differencecomponent.

A fourteenth aspect of the present invention is directed to the imageprocessing device of the twelfth aspect, further including a recordbutton, wherein when the record button is pressed, themulti-pixel-resolution image pick-up section records an image, and thelow-resolution video data and the resolution increasing parameter arerecorded.

A fifteenth aspect of the present invention is directed to the imageprocessing device of the twelfth aspect, further including a windowposition updating section for detecting a motion of an object candidateregion in the low-resolution video data and moving a position of thewindow region according to the detected motion of the object candidateregion.

A sixteenth aspect of the present invention is directed to the imageprocessing device of the fifteenth aspect, wherein the window positionupdating section detects the object candidate region based on afrequency analysis of an image.

A seventeenth aspect of the present invention is directed to the imageprocessing device of the twelfth aspect, wherein themulti-pixel-resolution image pick-up section obtains themulti-pixel-resolution video data while the multi-pixel-resolution videodata is split into a diffuse reflection component and a specularreflection component.

An eighteenth aspect of the present invention is directed to the imageprocessing device of the seventeenth aspect, wherein themulti-pixel-resolution image pick-up section splits a taken color imageinto a plurality of primary color components and splits one of theprimary color components into a specular reflection component and adiffuse reflection component, and wherein a diffuse reflection componentand a specular reflection component of the color image are obtained byusing the obtained specular reflection component and diffuse reflectioncomponent.

A nineteenth aspect of the present invention is directed to the imageprocessing device of the seventeenth aspect, wherein themulti-pixel-resolution image pick-up section estimates illuminatinglight used for recording, and the specular reflection component isobtained by using the estimated illuminating light.

A twentieth aspect of the present invention is directed to an imageprocessing device for performing a resolution increasing process byusing the low-resolution video data and the resolution increasingparameter recorded by the image processing device of the twelfth aspect,the image processing device including: a video input section for readingthe low-resolution video data; a resolution increasing parameter inputsection for reading the resolution increasing parameter; and aresolution increasing section for increasing a resolution of thelow-resolution video data read by the video input section by using theresolution increasing parameter read by the resolution increasingparameter input section.

A twenty-first aspect of the present invention is directed to an imageprocessing device, including: a multi-pixel-resolution image pick-upsection for obtaining multi-pixel-resolution video data by taking animage with a high resolution in a window region being a portion of anentire image and whose position changes every frame while taking animage with a low resolution outside the window region; a downwardresolution conversion section for performing an image filtering processon the high-resolution image in the window region to decrease aresolution of the high-resolution image; a multi-pixel-resolution videorecording section for recording the multi-pixel-resolution video data; aresolution increasing parameter learning section for learning aresolution increasing parameter by using the high-resolution image inthe window region and the output of the downward resolution conversionsection; and a resolution increasing parameter recording section forrecording the resolution increasing parameter.

A twenty-second aspect of the present invention is directed to the imageprocessing device of the twenty-first aspect, wherein themulti-pixel-resolution video recording section records themulti-pixel-resolution video data while the multi-pixel-resolution videodata is split into low-resolution video data and differential video datarepresenting a difference between the multi-pixel-resolution video dataand the low-resolution video data.

A twenty-third aspect of the present invention is directed to an imageprocessing device, including: a multi-pixel-resolution image pick-upsection for obtaining multi-pixel-resolution video data by taking animage with a high resolution in a window region being a portion of anentire image and whose position changes every frame while taking animage with a low resolution outside the window region; and amulti-pixel-resolution video recording section for recording themulti-pixel-resolution video data while the multi-pixel-resolution videodata is split into low-resolution video data and differential video datarepresenting a difference between the multi-pixel-resolution video dataand the low-resolution video data.

A twenty-fourth aspect of the present invention is directed to an imageprocessing device for obtaining a resolution increasing parameter byusing multi-pixel-resolution video data recorded by the image processingdevice of the twenty-third aspect, the image processing deviceincluding: a video separation section for reading themulti-pixel-resolution video data and separating the low-resolutionvideo data therefrom and for obtaining the high-resolution image in thewindow region by using the differential video data; a resolutionincreasing parameter learning section for learning a resolutionincreasing parameter by using the high-resolution image in the windowregion and the low-resolution video data; and a resolution increasingparameter recording section for recording the resolution increasingparameter.

A twenty-fifth aspect of the present invention is directed to an imagefile format for video data, wherein: for each frame, a resolution in awindow region being a portion of an entire image is higher than thatoutside the window region; a position of the window region changes everyframe.

A twenty-sixth aspect of the present invention is directed to the imagefile format of the twenty-fifth aspect, wherein a diffuse reflectioncomponent and a specular reflection component are stored separately.

A twenty-seventh aspect of the present invention is directed to an imagefile format representing video data in which a resolution in a windowregion being a portion of an entire image is higher than that outsidethe window region in each frame, wherein data of the image file formatincludes first compressed data obtained by compressing low-resolutionvideo data having the resolution outside the window region and secondcompressed data obtained by compressing differential video data, thedifferential video data representing the difference between the videodata and the low-resolution video data, and wherein a position of thewindow region changes every frame.

Preferred embodiments of the present invention will now be described indetail with reference to the drawings.

First Embodiment

FIGS. 1 and 2 are flow charts showing an image processing methodaccording to a first embodiment of the present invention. FIG. 1 shows arecording process of recording a scene, and FIG. 2 shows a resolutionincreasing process of reproducing and displaying the recorded videodata. The process of FIG. 1 and that of FIG. 2 correspond to the videorecording process and the reproduction process of watching the recordedvideo on a display, respectively.

The recording process of FIG. 1 first obtains and stores alow-resolution image LF(t) at time t (S11). At the same time, theprocess obtains a high-resolution image HWF(t) in a window region beinga portion of the image frame (S12). Since the high-resolution imageHWF(t) is only used in a later process, it does not need to be stored asvideo data. Based on these images LF(t) and HWF(t) of differentresolutions, the process learns and stores a resolution increasingparameter P(t) as a resolution conversion rule (S13). Then, the windowregion is moved to another position (S14). If the recording has not beencompleted, control proceeds to the next process at time t+1 (S16). Ifthe recording has been completed, the process ends (S15).

In step S13, the process obtains the texture feature for each of thelow-resolution image LF(t) and the high-resolution image HWF(t), andparameterize the correlation between the texture feature quantities. Theparameter P(t) is herein referred to as the “resolution increasingparameter”. Essentially, the parameter is a function or table as shownbelow for outputting a high-resolution texture TFH in response to aninput low-resolution texture TFL.

TFH=Function(TFL,t)  Exp. 1

The resolution increasing parameter is used as the resolution conversionrule.

In the resolution increasing process of FIG. 2, the process obtainslow-resolution video data LF(t) recorded in the recording process, andthe resolution increasing parameter P(t) learned in the recordingprocess (S21). Then, the process obtains a frame of video data (S22),and performs the resolution increasing process for that frame by usingthe resolution increasing parameter (S23). If the process is completedfor all the frames, the process ends (S24). Otherwise, the processproceeds to the next frame (S25).

In step S23, it is preferred that the resolution increasing process isperformed by using not only the resolution increasing parameter learnedin that frame but also those learned in at least one frame before orafter the current frame. This is because the resolution increasingparameter for the current frame is learned only for a window regionbeing a portion of the image, and may not therefore be sufficient forincreasing the resolution of the entire image. For example, the processmay use a resolution increasing parameter P(t′) for a period of timefrom t−T to t+T, where T is a predetermined amount of time, as shown inthe expression below.

t−T≦t′≦t+T  Exp. 2

FIG. 3 conceptually shows an example of a recording process and aresolution increasing process according to the present embodiment. InFIG. 3, each frame of the low-resolution video data LF(t) is an8×8-pixel image, and is divided into four window regions WD1 to WD4. Theposition of the window region changes every frame. In the illustratedexample, the window region is gradually moved in the vertical scanningdirection, i.e., from the top down to the bottom position and then backup to the top position. While the shape and the movement of the windowregion are simple in the illustrated example for ease of understanding,they are not limited to those of the illustrated example.

Within the window region (WD1 to WD4), the image is recorded with ahigher resolution than outside the window region to obtain thehigh-resolution image HWF(t). It is assumed herein that the image isrecorded in the window region with a pixel resolution 2×2 times as highas that outside the window region. The process may obtain video datawhere different pixel resolutions coexist in the same frame by recordingthe image with a high resolution within the window region and with a lowresolution outside the window region as will be described later. Suchvideo data is herein referred to as “multi-pixel-resolution video data”.

With such multi-pixel-resolution video data, a low-resolution image isrecorded outside the window region, but only a high-resolution image isrecorded within the window region. Note however that it is not necessaryto re-record a low-resolution image in the window region, which can beproduced by performing an image filtering process such as sub samplingor averaging on the high-resolution image. The modeling can be made morerealistic by taking into consideration the optical system and theaperture shape of the image pick-up device. For example, the conversioncan be done by applying a PSF (Point Spread Function) to thehigh-resolution image as a convolution calculation with a Gaussianfunction, and then integrating the obtained result over a range of theimage pick-up device region having a two-dimensionally constant width.

At the current point in time t, four areas AR1 to AR4 (arrangedvertically with one another) of the image correspond to the windowregion WD1 at time t−2, the window region WD2 at time t−1, the windowregion WD3 at time t, and the window region WD4 at t−3, respectively.Thus, at time t, the resolution increasing parameters for the areas AR1to AR4 have already been learned at times t−2, t−1, t and t−3,respectively. With a video data frame rate of 1/30 (sec), the length oftime from t−3 to t is only about 1/10 sec. Therefore, unless there is avery sudden change of scene, substantially the same feature informationis obtained in each of these frames. Thus, it is effective to use theresolution increasing parameters from time t−3 to time t.

Moreover, any low-resolution image region at time t is always recordedwith a high resolution at:

time t−1 or t+3;

time t−2 or t+2; or

time t−3 or t+1.

Therefore, where resolution increasing parameters from t−3 to t+3 areused, the resolution increasing process is effective unless there is avery sudden change of scene within the time frame. This is equivalent tosetting T=3 in Expression 2 above.

Assume resolution increasing parameters P(t−3), P(t−2), . . . are givenat corresponding points in time as shown in FIG. 3. In the resolutionincreasing process, the resolution of each area in each frame of thelow-resolution video data can be increased by using the resolutionincreasing parameter of a frame that is close in time to the subjectframe and at which the window region corresponds to the position of thesubject area. For example, with an image LF(t) at time t, the resolutionincreasing parameters P(t−2), P(t−1), P(t) and P(t+1) can be used forthe areas AR1, AR2, AR3 and AR4, respectively. Alternatively, theresolution increasing parameter P(t+2) may be used for the area AR1, andthe resolution increasing parameter P(t+3) for the area AR2.

While FIG. 3 shows an example where the frame is divided into fourwindow regions, the entire image may be divided into n window regions(where n is an integer of two or more), wherein the window region ismoved so that the entire image is scanned across over n frames. Alsowith other methods, it is preferred that the window region is moved sothat the entire image is scanned across over a plurality of frames.

FIG. 4 is a flow chart showing an example of the process in step S13 ofFIG. 1. This is an exemplary method for converting an image to a texturefeature vector, wherein resolution increasing parameters are learnedusing the multi-resolution analysis based on the wavelet transform.First, the low-resolution image LF(t) is enlarged by interpolation so asto match the image size of the low-resolution image LF(t) with that ofthe high-resolution image HWF(t) (S131). Then, the two images LF(t) andHWF(t) are transformed by a wavelet transform using three-step scalingprocess to be described later (S132). Finally, the correlation betweenthe wavelet coefficients of the two images LF(t) and HWF(t) is storedfor each pixel position (S133). The correlation is the resolutionincreasing parameter P(t).

FIG. 5 is a flow chart showing an example of a process in step S23 ofFIG. 2. Herein, a resolution increasing process is performed by usingthe resolution increasing parameter P(t′) for a period of time. First,the low-resolution image LF(t) is enlarged by interpolation to match theimage size thereof with that of the high-resolution image HWF(t) (S231).Then, the low-resolution image LF(t) is wavelet-transformed (S232), andthe wavelet coefficient is replaced by using the resolution increasingparameter P(t′) (S233). Then, an inverse wavelet transform is performedto covert the image to a high-resolution image (S234). In step S233, aresolution increasing parameter from a different frame time may be usedfor a portion of the entire image as described above with reference toFIG. 3. Alternatively, resolution increasing parameters over a period oftime may be quantized into a single set, which can be used as theresolution increasing parameter.

FIG. 6 shows an example of the wavelet transform in step S132 of FIG. 4.Referring to FIG. 6, in the first stage (scale), an input image IN issubjected to a wavelet transform in the X direction and in the Ydirection to be decomposed into four component images H1H1, H1L1, L1H1and L1L1 of the same size as the input image IN. In the second stage(scale), the components other than the component H1H1, which is thehigh-frequency component both in the X and Y directions, are furtherdecomposed. Only the component L1L1 is again decomposed both in the Xdirection and in the Y direction, whereas the other components H1L1 andL1H1 are decomposed only in one direction, thus resulting in a total ofeight components. In the third stage (scale), the components other thanthe components H1H2, H2H1 and H2H2 are decomposed similarly, whereinonly the component L2L2 is decomposed both in the X and Y directions,whereas the other components are decomposed only in one direction, thusresulting in a total of 12 components. Those that are re-decomposed intotwo or four as the scale increases (those shown in dotted lines) can beproduced by subsequent synthesis.

Through the wavelet transform decomposition as shown in FIG. 6, eachpixel of the input image IN is decomposed into 16 components as shown inFIG. 7. A vector obtained by combining together these 16 components is atexture feature TF1 at a pixel position PP.

As described above, the low-resolution image and the high-resolutionimage are each converted to a texture feature, and the correlationbetween the texture feature quantities is learned so as to produce ananalysis code book and a reproduction code book. This process isdescribed in detail in, for example, Yoshito Abe, Hisakazu Kikuchi,Shigenobu Sasaki, Hiromichi Watanabe and Yoshiaki Saitoh “Edgeenhancement of images using multiresolution vector quantization”, IEICETransactions, Vol. J79A 1996/5 (pp. 1032-1040). In such a case, theparameterization in step S13 corresponds to the production of ananalysis code book and a reproduction code book. The resolutionincreasing parameter P(t) shown in Expression 1 above is calculated fromthis.

In the present embodiment, a high-resolution image is obtained only in awindow region being a portion of the image during the recordingoperation, and a resolution increasing parameter is produced by using animage feature such as a texture. Employment of such a process has thefollowing advantages.

Firstly, instead of obtaining the resolution increasing parameter in apreliminary learning process from different video data, the resolutionincreasing parameter can be learned for the object itself of video datato be later subjected to the resolution increasing process, whereby itis possible to realize a high-precision image process. Moreover, sincethe resolution increasing parameter is a function of time, if it islearned for each frame, it is possible to instantaneously obtain anoptimal resolution increasing parameter irrespective of thecharacteristics of an object appearing in the screen. Thus, inprinciple, there is no performance deterioration due to the differencein characteristics between the resolution increasing parameter obtainedby a learning process and video data whose resolution is actuallyincreased.

Secondly, what is used in the resolution increasing process is a featuresuch as the image texture, and has no direct correlation with theposition in the screen, whereby it is not substantially influenced bythe movement of the window region across the screen. Therefore, byappropriately moving the window to cover the entire image, it ispossible to sufficiently use information from the entire image.

While the recording process shown in FIG. 1 and the resolutionincreasing process shown in FIG. 2 are typically performed sequentiallyin this order, the recording process and the resolution increasingprocess may be performed in parallel in time in a case where the camerarecording process and the transmission/display process are performed atthe same time as in a live broadcast.

In the present embodiment, only the low-resolution video data isrecorded in the recording process, whose resolution is then increased ina resolution increasing process. Alternatively, for example,multi-pixel-resolution video data as shown in FIG. 3 may be recorded,whose resolution can later be increased in a resolution increasingprocess.

Second Embodiment

A second embodiment of the present invention is based on the firstembodiment as described above, and is directed to a specific deviceconfiguration. In this embodiment, the recording process is performed byusing a camcorder including a multi-pixel-resolution image pick-upsection capable of recording multi-pixel-resolution video data asdescribed above. The multi-pixel-resolution image pick-up section isrealized by using an XY address type imaging device such as a CMOS-typeimage pick-up sensor.

FIG. 8 is a block diagram showing an exemplary configuration of an imageprocessing device of the present embodiment, wherein the presentinvention is applied to a camcorder. An image processing device 10 ofFIG. 8 includes a lens 101, a multi-pixel-resolution image pick-upsection 102, a low-resolution frame memory 103 for storing one frameLF(t) of low-resolution video data, a low-resolution video recordingsection 104 for recording low-resolution video data DLF, ahigh-resolution window memory 105 for storing the high-resolution imageHWF(t) in the window region in one frame, a downward resolutionconversion section 106 for performing an image process to convert thehigh-resolution image HWF(t) in the window region to a lower resolution,a resolution increasing parameter learning section 120 for learning theresolution increasing parameter P(t) by using the high-resolution imageHWF(t) in the window region, a resolution increasing parameter recordingsection 109 for recording a resolution increasing parameter DP, anoverall control section 110, and a record button 111. The resolutionincreasing parameter learning section 120 includes a texture conversionsection 107 for converting the high-resolution image HWF(t) to a texturefeature, an image size correction/texture conversion section 112 forcorrecting the pixel size of the low-resolution image and thenconverting it to a texture, and a texture learning section 108 forlearning the texture correlation. Since the multi-pixel-resolution imagepick-up section 102 herein reads out signals line by line, a subsampling circuit 115 for performing a horizontal pixel sub samplingprocess is provided between the multi-pixel-resolution image pick-upsection 102 and the low-resolution frame memory 103.

When the record button 111 is pressed, the overall control section 110sends out a record command to the multi-pixel-resolution image pick-upsection 102 and the low-resolution video recording section 104. When therecord command is received, the multi-pixel-resolution image pick-upsection 102 performs the recording process, wherein the low-resolutionimage LF(t) is stored in the low-resolution frame memory 103 for eachframe. Then, the low-resolution video recording section 104 producesvideo data by chronologically combining together the low-resolutionimages LF(t) stored in the low-resolution frame memory 103, and furtherperforms processes such as a data compressing process to obtain thelow-resolution video data DLF, which is recorded on a storage medium ora network.

Simultaneously with the record command, the overall control section 110sends out a learning signal to the texture learning section 108 and theresolution increasing parameter recording section 109. Thehigh-resolution image HWF(t) in the window region obtained by themulti-pixel-resolution image pick-up section 102 is stored in thehigh-resolution window memory 105, and is input to the texture learningsection 108 via the texture conversion section 107. The high-resolutionimage HWF(t) is input to the texture learning section 108 also via theimage size correction/texture conversion section 112 after beingsubjected to the downward resolution conversion by the downwardresolution conversion section 106. The texture learning section 108learns the resolution increasing parameter P(t), being a resolutionconversion rule, by associating together textures from images of thesame point in time but of different resolutions. The resolutionincreasing parameter recording section 109 stores the parameter as theresolution increasing parameter DP, or sends out the parameter to anetwork.

The output of the downward resolution conversion section 106 is alsosupplied to the low-resolution frame memory 103. This is for filling thewindow region portion, which is missing in the low-resolution imageLF(t), because the multi-pixel-resolution image pick-up section 102obtains only the high-resolution image but does not read out thelow-resolution image in the window region.

This operation is stopped when the record button 111 is released.

In the configuration of FIG. 8, sections from the multi-pixel-resolutionimage pick-up section 102 to the low-resolution video recording section104 can be implemented by hardware, for example, and the texturelearning section 108 and the resolution increasing parameter recordingsection 109 can be implemented primarily based on a CPU or GPU (GraphicProcessing Unit).

FIG. 9 is a conceptual diagram showing an exemplary configuration of themulti-pixel-resolution image pick-up section 102. The configuration ofFIG. 9 employs an ordinary three-chip optical system for color imagerecording used in a camcorder. With a three-chip system, a dichroicprism 131 splits light into three wavelength bands of red (R), green (G)and blue (B). These color bands are assigned multi-pixel-resolutionimage pick-up devices 132R, 132G and 132B, respectively. A signalprocessing circuit 133 processes the outputs from themulti-pixel-resolution image pick-up devices 132R, 132G and 132B toproduce the luminance signal Y and the color difference signals Cr andCb. In this process, a low-resolution signal obtained by a “skipread-out operation” to be described later and a high-resolution signalobtained by a “progressive read-out operation” also to be describedlater are output separately.

Video data is recorded in the YCrCb format. For texture learning, theluminance-color difference signals may be all used in the YCrCb format,or only the luminance Y signal may be used. However, where theresolution increasing factor is greater than 4×4, the resolutionincreasing process using only the luminance Y results in an insufficientimage quality in view of the frequency response characteristics for theluminance component and those for the color component of the humanvisual system, whereby it is necessary to perform the resolutionincreasing process separately for the color difference CrCb. Therefore,for a factor of 4×4 or more, three signal components of red R, green Gand blue B may be used for the texture learning. In the configuration ofFIG. 9, red, green and blue are assigned the multi-pixel-resolutionimage pick-up devices 132R, 132G and 132B, respectively, and the videodata recording format is set to the conventional luminance-colordifference space, whereby the luminance signal Y and the two colordifference signals Cr and Cb are used independently for the texturelearning.

FIGS. 10A to 10D show an example of read-out pixel positions for amulti-pixel-resolution image pick-up device. In FIGS. 10A to 10D,hatched pixels are pixels to be read out. While a practical imagepick-up device has about 2000×3000 pixels, the figure shows a16×16-pixel two-dimensional sensor, i.e., 16 vertical lines and 16horizontal lines. FIGS. 10A to 10D show four frames that are contiguousin time.

As can be seen from FIGS. 10A to 10D, the 256 (=16×16) pixel region isdivided into a region where all pixels are to be read out (correspondingto the window region in which the high-resolution image is obtained) andanother region where a skip read-out operation is performed. The upwardmovement of the window region as shown in FIGS. 10A to 10D is repeated,in which each cycle is made up of the four frames of FIGS. 10A to 10D.

The method for obtaining a low-resolution image outside the windowregion may be any of various methods applicable. In the illustratedexample, each read-out pixel position of the low-resolution image is atabout the center of one of 16 virtual blocks into which the entire imageis equally divided. This method produces a low-resolution image that hasbeen sub sampled to ¼ in the horizontal (H) direction and in thevertical (V) direction. The vertical line numbers to be read out in theskip read-out operation are (7,11,15) in FIG. 10A, (3,11,15) in FIG.10B, (3,7,15) in FIG. 10C and (3,7,11) in FIG. 10D.

As described above, the window region where all lines are read out isregularly moved in the vertical direction by four lines, whereas threefixed lines in the vertical directions are read out outside the windowregion. With this method, it is possible to perform themulti-pixel-resolution recording operation without reading out the samepixel twice. This read-out method is used in the present embodiment.

FIGS. 11A to 11D show another example of read-out pixel positions usedby the multi-pixel-resolution image pick-up device. The example shown inFIGS. 11A to 11D differs from that shown in FIGS. 10A to 10D in themethod for reading out the low-resolution image. With this method, a¼-sub sampled low-resolution image is read out at a different positionfor each frame, independently of the operation of reading out all linesin the window region. The purpose of this method is to equalize thesampling in the time and space domains, and allows for some pixels to beread out more than once. The vertical line numbers to be read out in theskip read-out operation are (1,5,9,13) in FIG. 11A, (2,6,10,14) in FIG.11B, (3,7,11,15) in FIG. 11C and (4,8,12,16) in FIG. 11D. Pixels shownin solid black are those that are read out more than once. With thismethod, the low-resolution video data can be obtained only with pixelsread out in the skip read-out operation. Therefore, it is not necessaryto supply the pixel values from the downward resolution conversionsection 106 to the low-resolution frame memory 103 in order to fill thewindow region mixing in the low-resolution image.

Configuration of Multi-Pixel-Resolution Image Pick-Up Device

FIG. 12 is a circuit block diagram showing an exemplary configuration ofthe multi-pixel-resolution image pick-up device. FIG. 12 shows, as anexample, a two-dimensional sensor with 16 vertical lines and 16horizontal lines, capable of implementing a read-out method as shown inFIGS. 11A to 11D. In the following description, terms based on lines areused, and a process of reading out a portion of the screen with a highresolution is referred to as the “progressive read-out operation”whereas a process of reading out the screen with a low resolution isreferred to as the “skip read-out operation”.

Generally, the image pick-up device includes an image pick-up section201 and a storage section 202. The image pick-up section 201 includesimaging pixels 211 capable of photoelectric conversion of the incidentlight, which are arranged in a two-dimensional arrangement (array). Thestorage section 202 includes image-storing pixels 221 including alight-blocking portion for blocking the incident light, which arearranged in a two-dimensional arrangement, each image-storing pixel 211corresponding to one of the imaging pixels 211. The image pick-upsection 201 and the storage section 202 are connected together via asignal line p2str for transferring signal charges from the image pick-upsection 201 to the storage section 202.

Around the image pick-up section 201, the image pick-up device includesa read-out shift register 203 for successively shifting the read-outpulse line by line, and a reset shift register 204 for successivelyshifting the reset pulse line by line. The read-out pulse and the resetpulse are sent to each imaging pixel 211 of the image pick-up section201 via a read-out pulse supply line rd-p and a reset pulse supply linerst-p, respectively. The imaging pixel 211 receiving the read-out pulseoutputs the signal charge, and the imaging pixel 211 receiving the resetpulse resets the signal charge.

Around the storage section 202, the image pick-up device includes a skipscanning shift register 205 for outputting a read-out pulse used in the“skip read-out operation”, a progressive scanning shift register 206 foroutputting a read-out pulse used in the “progressive read-outoperation”, a selector 207 for selectively supplying the output from theskip scanning shift register 205 or the output from the progressivescanning shift register 206 to the storage section 202, and a read shiftregister 208 used when transferring the signal charge from the imagepick-up section 201 to the storage section 202. The read-out pulseoutput from the selector 207 is supplied to each image-storing pixel 221of the storage section 202 via a read-out pulse supply line rd-s. Thetransfer pulse output from the read shift register 208 is supplied toeach image-storing pixel 221 of the storage section 202 via a transferpulse supply line trn.

The progressive scanning shift register 206 generates a pulse forpartially reading out the high-resolution image (obtained by reading outall lines) in each frame (e.g., outputting a ¼ screen in each frame toform one full screen over four frames). The skip scanning shift register205 generates a pulse for obtaining in each frame one screen of thelow-resolution image obtained by the skip read-out operation. The skipscanning shift register 205 will later be described in detail withreference to FIG. 15.

The image pick-up device also includes a group of select transistors209, a horizontal shift register 210 and an output amplifier 211. Thesignal charges stored in the image-storing pixels 221 of the storagesection 202 are output to the outside via signal output lines sig-outand through the group of select transistors 209 and the output amplifier211 based on the order in which they are selected by the horizontalshift register 210.

A timing generation circuit 212 supplies pulses to the read-out shiftregister 203, the reset shift register 204, the skip scanning shiftregister 205, the progressive scanning shift register 206, the selector207, the read shift register 208, the horizontal shift register 210 anda reset pulse supply line rst-s. Note however that the timing generationcircuit 212 may be separately provided on a different chip from theimage pick-up device.

The signal line p2str, the read-out shift register 203, the read-outpulse supply line rd-p, the read shift register 208 and the transferpulse supply line trn together form a transfer section. The signaloutput line sig-out, the skip scanning shift register 205, theprogressive scanning shift register 206, the selector 207 and theread-out pulse supply line rd-s together form an output section.

FIG. 13 shows an exemplary configuration of the imaging pixel 211. Theconfiguration of FIG. 13 is an orthodox 3-transistor configuration.Specifically, the imaging pixel 211 includes a photodiode PD forgenerating a signal charge in response to light, a read-out transistorTR1 whose gate is connected to the read-out pulse supply line rd-p foroutputting the signal charge to the signal line p2str, a source followertransistor TR2, and a reset transistor TR3 whose gate is connected tothe reset pulse supply line rst-p for resetting the stored signalcharge.

FIG. 14 shows an exemplary configuration of the image-storing pixel 221.The configuration of FIG. 14 includes four transistors and onecapacitor. Specifically, the image-storing pixel 221 includes atransistor TR4 whose gate is connected to the transfer pulse supply linetrn, a storage capacitor C-str for storing a signal charge transferredfrom the imaging pixel 211 through the signal line p2str and thetransistor TR4, a transistor TR5 whose gate is connected to the read-outpulse supply line rd-s for reading out the signal charge stored in thestorage capacitor C-str to the signal output line sig-out, a transistorTR6 whose gate is connected to the reset pulse supply line rst-s forresetting the storage capacitor C-str to the GND level, and a sourcefollower transistor TR7.

FIG. 15 shows an exemplary configuration of the skip scanning shiftregister 205. In FIG. 15, the skip scanning shift register 205 includesa skip line designation register 251, a group-of-lines scanning shiftregister 252, a D latch 253, etc. The details of the operation thereofwill be described later.

FIG. 16 is an operation sequence diagram illustrating the outline of aseries of operations, i.e., the image-recording operation, the transferoperation and the read-out operation, by the multi-pixel-resolutionimage pick-up device shown in FIGS. 12 to 15. FIG. 16 shows the generaloperation over a series of four frames (note however that the thirdframe is omitted due to limitations of space).

In the first frame, after the exposure time T1 for a line PDR1 in theimage pick-up section 201 is completed, the signal charges on the linePDR1 are transferred at once to pixels along the corresponding line STR1in the storage section 202 during the following transfer period T11. Thetransferred signal charges are stored in the storage capacitors C-str inthe image-storing pixel 221 of corresponding horizontal positions. Then,after the exposure time T2 for a line PDR2 is completed, the signalcharges on the line PDR2 are transferred at once to pixels along thecorresponding line STR2 in the storage section 202 during the followingtransfer period T21. The transferred signal charges are stored in thestorage capacitors C-str in the image-storing pixel 221 of correspondinghorizontal positions.

The read-out/transfer operation is performed successively for 16 linesPDR1 to PDR16 during the high-speed V transfer period Tp2str.Specifically, the signal charges obtained by photoelectric conversion bythe photodiodes PD in the imaging pixels 211 during an exposure time aretransferred in the high-speed V transfer period Tp2str from the imagepick-up section 201 to the storage section 202, and are stored in thestorage section 202.

The signal charges stored in the storage section 202 in the first frameare read out to the outside during the horizontal read-out period in thenext, second frame. In the second frame, the progressive read-outoperation is performed for the four lines STR5 to STR8, and the skipread-out operation is performed for the four lines STR2, STR6, STR10 andSTR14. The other lines are not read out. As the reset pulse is appliedto the reset pulse supply line rst-s in the storage section resettingperiod, the storage capacitors C-str in all the image-storing pixels 221of the storage section 202 are all reset to the GND level.

A similar scanning operation is performed in the subsequent frames. Inthe fourth frame, the progressive read-out operation is performed forthe four lines STR13 to STR16, and the skip read-out operation isperformed for the four lines STR4, STR8, STR12 and STR16. The otherlines are not read out. As the reset pulse is applied to the reset pulsesupply line rst-s in the storage section resetting period, the storagecapacitors C-str in all the image-storing pixels 221 of the storagesection 202 are all reset to the GND level.

FIG. 16 shows an example where in each frame, a series of four lines areread out in a progressive read-out operation and four non-contiguouslines selected every four lines are read out in a skip read-outoperation. However, the number of lines to be read out is not limited tothis, and the number of lines to be read out in the progressive read-outoperation does not need to be the same as the number of lines to be readout in the skip read-out operation. For example, the progressiveread-out operation may be performed for a series of n lines, and theskip read-out operation may be performed every m lines. Moreover, whilethe line STR6 is read out more than once in the second frame for examplefor the sake of simplicity, a line does not need to be read out morethan once.

FIG. 17 is a timing diagram showing an operation in the high-speed Vtransfer period Tp2str. Referring to FIG. 17, after the exposure time T1is completed for the line PDR1 in the first frame, a read-out pulse issupplied to the read-out pulse supply line rd-p. The read-out pulse isapplied to the gate of the transistor TR1 in the imaging pixel 211 ofFIG. 13, and a signal potential corresponding to the signal charge ofthe photodiode PD is output to the signal line p2str via the sourcefollower transistor TR2. In the line STR1, when a transfer pulse issupplied to the transfer pulse supply line trn, the transfer pulse isapplied to the gate of the transistor TR4 in the image-storing pixel 221of FIG. 14, and a signal charge is transferred from the signal linep2str to the storage capacitor C-str via the transistor TR4.

After the read-out pulse is supplied, a reset pulse is supplied to thereset pulse supply line rst-p. The reset pulse is applied to the gate ofthe transistor TR3 in the imaging pixel 211 of FIG. 13, therebyresetting the photodiode PD.

With such an operation, the signal charges of the imaging pixels 211 ineach of the lines PDR1 to PDR16 in the image pick-up section 201 are alltransferred to the image-storing pixels 221 of the corresponding one ofthe lines STR1 to STR16 in the storage section 202.

FIG. 18 is a timing diagram showing the operation in the horizontalread-out period and the storage section resetting period. In FIG. 18,the clock CK is supplied from the timing generation circuit 212. In thehorizontal read-out period of the first frame, the shift clock CK-H,which is cut out from the clock CK by the selection signal sel-H/L, isgiven to the progressive scanning shift register 206. Receiving theshift clock CK-H and the shift data Data-H (not shown) supplied from thetiming generation circuit 212, the progressive scanning shift register206 outputs the read-out pulse toward the lines STR1 to STR4. In thehorizontal read-out period of the second frame, the progressive scanningshift register 206 outputs the read-out pulse toward the lines STR5 toSTR8.

As shown in FIG. 15, the skip scanning shift register 205 includes theskip line designation register 251 for determining the lines to beskipped in the frame, and the group-of-lines scanning shift register 252for scanning a plurality of lines with the same phase. First, the skipline designation register 251 receives the clock CK-L1 from the timinggeneration circuit 212 and skip line selection data Data-L1, and bringsthose of the outputs L1 to L16 corresponding to lines to be skipped inthe frame to “H”. In the first frame, the outputs L1, L5, L9 and L13 areturned “H”.

Next, the group-of-lines scanning shift register 252 receives the datashift clock CK-L2 from the timing generation circuit 212 and the dataData-L2. It is assumed herein that the data Data-L2 is data whose periodis equal to four cycles of the data shift clock CK-L3 and whose level is“H” during one cycle (corresponding to four pulses of the data shiftclock CK-L2) and “L” during the following three cycles. Thus, theoutputs LT1 to LT16 of the D latch 253 are as shown in FIG. 18.

FIG. 19 is a timing diagram showing the output of the selector 207. Theselector 207 selects the output of the progressive scanning shiftregister 206 when the selection signal sel-H/L is at “H”, and selectsthe output of the skip scanning shift register 205 when the selectionsignal sel-H/L is at “L”. Thus, as a result of the series of operationsshown in FIG. 18, outputs as shown in FIG. 19 are obtained for the linesSTR1 to STR16.

The number of lines to be skipped can be changed by changing the dataData-L2, the data shift clock CK-L2 and the data shift clock CK-L3,which are output from the timing generation circuit 212. The number ofconsecutive lines to be scanned in the progressive scanning operationcan be changed by changing the “H” period of the selection signalsel-H/L, which is also output from the timing generation circuit 212.Thus, the number of lines to be skipped and the number of consecutivelines to be scanned in the progressive scanning operation are notdictated by the circuit configuration, but the mode of operation canfreely be set, allowing for the operation with a high degree of freedom,in the present embodiment.

The operation of reading out one line in the horizontal read-out periodis performed as follows. The horizontal shift register 210 receives aone-horizontal-period selection signal HSEL and the horizontal transferclock Hck supplied from the timing generation circuit 212, and inresponse supplies pulses to the transistors C1 to C16 of the group ofselect transistors 209 one after another. As a result, the signalcharges stored in the storage capacitors C-str of the image-storingpixels 221 are successively transferred from the lines STR1 to STR16 asthey are selected by the output of the selector 207 to an externalsignal processing circuit (not shown) via the output amplifier 211, thuscompleting the read-out operation.

Referring back to the configuration of FIG. 8, the pixel valuesprogressively read out from the multi-pixel-resolution image pick-upsection 102 are temporarily stored in the high-resolution window memory105 as the high-resolution image HWF(t). The pixel values read out in askip read-out operation from the multi-pixel-resolution image pick-upsection 102 are sub sampled to ¼ in the H direction by the sub samplingcircuit 115 and then temporarily stored in the low-resolution framememory 103 as one frame screen. The sub sampling process is realized byallowing only pixel values of the pixel numbers (2,6,10,14) in the Hdirection to pass while discarding the other pixel values in the exampleof FIGS. 10 and 11.

With the read-out operation of FIGS. 10A to 10D as it is, thelow-resolution image in the window region of the frame will be dropped.In view of this, the downward resolution conversion section 106 performsan image process such as a sampling process for the high-resolutionimage HWF(t), and stores the resulting image at the correspondingposition in the low-resolution frame memory 103. With the example ofFIGS. 10A to 10D, for example, particular pixels at the positions ofcoordinates (V,H)=(2,3), (6,3), (10,3) and (14,3) can be sampled withthe bottom side of the window region being the reference point (origin)in the V direction.

The low-resolution image LF(t), which has been sub sampled to ¼×¼ andstored in the low-resolution frame memory 103, is successively recordedby a low-resolution video data recording section 104 frame by frame. Theprocess may or may not employ a video data compression scheme known inthe art.

The luminance component of the high-resolution image HWF(t) stored inthe high-resolution window memory 105 is input to the texture conversionsection 107, and is converted to a luminance image texture feature by amulti-resolution conversion such as wavelet transform. The luminancecomponent of the low-resolution image output from the downwardresolution conversion section 106 is input to the image sizecorrection/texture conversion section 112. In the image sizecorrection/texture conversion section 112, the low-resolution image isonce converted back to the same number of pixels as the high-resolutionimage and then subjected to a multi-resolution conversion such as awavelet transform, thereby obtaining a luminance image texture feature.The method for correcting the image size may be of any type, includingbilinear interpolation, bicubic interpolation, and the like. The imagesize correction method used in this process is used as a pre-processwhen performing the resolution increasing process on the low-resolutionimage for displaying the video data with a high resolution.

FIG. 20 conceptually shows the texture conversion process. Referring toFIG. 20, an image 2001 is a high-resolution image in the window region,which is in a square shape in the illustrated example. An image 2002 isa wavelet coefficient image made up of a plurality of levels obtained byperforming the wavelet transform operation on the high-resolution image2001. An image 2003 is a low-resolution image, an image 2004 is an imageobtained by performing a pixel size correction process on the image 2003to match the number of pixels with that of the high-resolution image2001, and an image 2005 is a wavelet coefficient image obtained from theimage 2004. Where the image 2001 and the image 2004 are referred to as asharpened image and a blurred image, respectively, reference numerals2006 and 2007 are texture feature quantities having a 16^(th)-ordercomponent at the pixel position PP in the sharpened image 2001 and inthe blurred image 2004, respectively.

During the learning (video recording) process, the process learns thecorrelation between the texture feature quantities 2006 and 2007 tothereby obtain, as the resolution increasing parameter, a conversionrule for converting the texture feature 2007 of a lower resolution tothe texture feature 2006 of a higher resolution. Then, in the resolutionincreasing process, the received low-resolution image 2003 is subjectedto a pixel size correction process to obtain the blurred image 2004, towhich the learned resolution increasing parameter is applied to therebyobtain the high-resolution image 2001. In the illustrated example, theresolution increasing process is performed by using the image, which isthe source of the learning process, and therefore an ideal resolutionincreasing process is realized. In practice, the process deals withunlearned images, and it is not always possible to restore an idealhigh-resolution image. Nevertheless, in the present invention, theresolution increasing parameter is learned in the window region,accounting for a portion of the object, during the video recordingprocess, whereby it is possible to realize a resolution increasingprocess of a much higher precision than a method in which the resolutionincreasing parameter is obtained by leaning a similar object in advance.

The texture learning section 108 learns the correlation between thetexture conversion section 107 and the texture feature output from theimage size correction/texture conversion section 112 to create ananalysis code book and a reproduction code book and produce theresolution increasing parameter. The method will now be described withreference to FIG. 21.

It is assumed herein that the blurred image and the sharpened image areeach made up of 100 pixels. Each pixel of the blurred image and thesharpened image is converted to a multi-resolution vector, to therebyobtain multi-resolution vectors U1 to U100 and V1 to V100. The vectorsU1 to U100 and the vectors V1 to V100 are in such a relationship thateach of them is of the same pixel position as its counterpart.Therefore, the code book can be produced so that when a vector U isinput, a corresponding vector V is output. In practice, however, themulti-resolution vectors are classified into representative vectors by avector quantization process.

In the example of FIG. 21, the vectors U are quantized into two types2101 and 2102, and the vectors V into two types 2103 and 2104. Aquantization index of the analysis code book or the reproduction codebook is the number assigned to a set of vectors obtained by thequantization process. Looking up in the code book means to input avector number V to obtain a number 1 or 2, which designates a set ofvectors obtained by the quantization process. A quantized set 2103 isassigned a representative vector Z1, and the quantized set 2104 arepresentative vector Z2. A representative vector for a quantized set iscalculated by, for example, taking an average value or a representativevalue of the vectors belonging to that quantized set. In the presentinvention, the code book as described above is calculated for each frameas a function of time t, thus obtaining the resolution increasingparameter P(t).

Next, the process produces an analysis code book IG for outputting aquantization index (1 or 2) in response to a vector number, and areproduction code book IF for outputting a reproduction vector for aninput quantization index (1 or 2). By using the produced analysis codebook and reproduction code book in combination with each other, it ispossible to convert a multi-resolution vector of a blurred image to thatof a sharp image.

The obtained analysis code book and reproduction code book are stored asthe resolution increasing parameter DP by the resolution increasingparameter recording section 109.

FIG. 22 shows a configuration of an image display section of the presentembodiment. The configuration of FIG. 22 has a function of displaying,with a high resolution, video data recorded by an image processingdevice such as a camcorder of FIG. 8. For example, the image displaysection is implemented as a display section of the camcorder or providedin a portable display device or a household large-screen TV set. Animage display section 30 of FIG. 22 includes a video input section 301,an image size correction/texture conversion section 302, a texturereplacing section 303, a texture inverse conversion section 304, aresolution increasing parameter input section 305, an overall controlsection 306 and a display 307. The image size correction/textureconversion section 302, the texture replacing section 303 and thetexture inverse conversion section 304 together form a resolutionincreasing section 310.

First, the low-resolution video data DLF and the resolution increasingparameter DP recorded by the image processing device 10 of FIG. 8 areinput to the image display section 30 via a network or via various othermedia. The overall control section 306 sends a command to a video datainput section 301, and the video data input section 301 receiving thecommand expands and reads out the compressed low-resolution video dataDLF, for example. It is assumed herein that the video data is ordinarycolor video data, and therefore a YCrCb luminance-color differencesignal is produced. The read-out image is of a low resolution whosepixel size is ¼×¼ of the original high-resolution image. Therefore, theimage size correction/texture conversion section 303 then enlarges theimage size to 4×4 to obtain a blurred image, and performs a textureconversion process independently on the luminance component Y and thecolor difference component CrCb. This operation is similar to the imagesize correction/texture conversion section 112 of FIG. 8, and will notbe further described below.

Next, the overall control section 306 sends a command to the resolutionincreasing parameter input section 305, and the resolution increasingparameter input section 305 receiving the command reads out theresolution increasing parameter DP and inputs the parameter to thetexture replacing section 303. By using the analysis code book and thereproduction code book described in the resolution increasing parameterin combination with each other, the texture replacing section 303converts a multi-resolution vector (texture feature) representing thetexture of a blurred image to a multi-resolution vector representing thetexture of a sharp image. Then, the texture inverse conversion section304 converts the sharp image texture feature to a resolution-increasedluminance image. Then, the resolution-increased luminance Y image andthe original color difference CrCb image are input to the display 307,and the high-resolution image is displayed as video data.

The above description is based on the principle that color video data isseparated into luminance/color difference components YCrCb, and theresolution of each component is increased independently. Note howeverthat the present invention is not limited to this, but may employ amethod in which color video data is separated into RGB components, whichare subjected to the resolution increasing process independently, or amethod in which a black-and-white image having only the luminancecomponent is subjected to the resolution increasing process.

In the present embodiment, it is possible to produce and display ahigh-resolution video data from a recorded low-resolution video data, asdescribed above, whereby the amount of information needed to be storedwhen the camera records the video is small, and there is no need to usea power-consuming super speed imaging device for recordinghigh-resolution video data. Thus, the present invention is applicable toa super-small video camera having limitations on the amount of powerthat can be consumed. As a result, the present invention providessignificant advantages that video data recorded by a super-small videocamera can be displayed on an enlarged scale with a high image qualityon a large-screen display, and that a portion of the image of interestcan be further enlarged.

Third Embodiment

FIG. 23 is a block diagram showing an exemplary configuration of animage processing device according to a third embodiment of the presentinvention, wherein the present invention is applied to a camcorder as inFIG. 8. The configuration differs from that of FIG. 8 in that thelow-resolution frame memory 103 is omitted, and a multi-pixel-resolutionvideo recording section 401 for recording multi-pixel-resolutioncompressed video data DMC is provided instead of the low-resolutionvideo recording section 104 for recording the low-resolution video dataDLF. Thus, while video data with a lowered resolution is recorded in thesecond embodiment, image data obtained by the multi-pixel-resolutionimage pick-up section 102 is stored as it is in the present embodiment.By effectively using the obtained multi-pixel-resolution video datawithout lowering its resolution, the present embodiment aims to improvethe performance of the resolution increasing process.

When the record button 111 is pressed, the overall control section 110sends out a record command to the multi-pixel-resolution image pick-upsection 102 and the multi-pixel-resolution video recording section 401.When the record command is received, the multi-pixel-resolution imagepick-up section 102 performs the recording process, wherein themulti-pixel-resolution video data F(t) is sent to themulti-pixel-resolution video recording section 401. Receiving themulti-pixel-resolution video data F(t), the multi-pixel-resolution videorecording section 401 performs a process such as a compression processto be described later, and records the resulting data as themulti-pixel-resolution compressed video data DMC on a storage medium ora network.

Simultaneously with the record command, the overall control section 110sends out a learning signal to the texture learning section 108 and theresolution increasing parameter recording section 109. Thehigh-resolution image HWF(t) in the window region obtained by themulti-pixel-resolution image pick-up section 102 is stored in thehigh-resolution window memory 105, and is input to the texture learningsection 108 via the texture conversion section 107. The high-resolutionimage HWF(t) is input to the texture learning section 108 also via theimage size correction/texture conversion section 112 after beingsubjected to the downward resolution conversion by the downwardresolution conversion section 106. The texture learning section 108learns the resolution increasing parameter P(t), being a resolutionconversion rule, by associating together textures from images of thesame point in time but of different resolutions. The resolutionincreasing parameter recording section 109 stores the parameter as theresolution increasing parameter DP, or sends out the parameter to anetwork.

FIG. 24 shows an image format of the multi-pixel-resolution compressedvideo data DMC. The process illustrated herein is performed by themulti-pixel-resolution video recording section 401. Themulti-pixel-resolution recorded video data F(t) recorded by themulti-pixel-resolution image pick-up section 102 includes portions ofdifferent pixel resolutions within one frame, and therefore cannotefficiently be compressed as video data. Therefore, there is firstproduced video data LF(t) whose resolution has been lowered entirely.This can be performed by an image filtering process as described abovein the second embodiment, for example. Next, the low-resolution videodata LF(t) is compressed to produce low-resolution compressed video dataCLF(t) as first compressed data. This can be done by using a videocompression method known in the art.

Then, differential video data DF(t) is produced.

DF(t)=F(t)−LF(t)  Exp. 3

The differential video data DF(t) has a differential value only in thewindow region where the high-resolution image has been obtained, withthe differential value being zero in other regions. Thus, thedifferential video data DF(t) has data over a small area. In addition,the data contains a lot of high-frequency components, thus resulting ina high compression efficiency in the quantization process. In view ofthis, the differential video data DF(t) is compressed separately toobtain the differential compressed video data CDF(t) as secondcompressed data. The two compressed video data CLF(t) and CDF(t) arecombined together in a single image format, which is recorded as themulti-pixel-resolution compressed data DMC.

FIG. 25 shows a configuration of an image display section of the presentembodiment. The configuration of FIG. 25 has a function of displaying,with a high resolution, video data recorded by an image processingdevice such as a camcorder of FIG. 23. For example, the image displaysection is implemented as a display section of the camcorder or providedin a portable display device or a household large-screen TV set. In FIG.25, like elements to those of FIG. 22 are denoted by like referencenumerals and will not be further described below. The configuration ofFIG. 25 differs from that of FIG. 22 in that the multi-pixel-resolutioncompressed video data DMC is received as the input, and theconfiguration accordingly includes a compressed video data separationsection 402 for separating the input multi-pixel-resolution compressedvideo data DMC into the differential video data DF(t) and thelow-resolution video data LF(t). Moreover, the configuration includes ahigh-resolution window producing section 403, a high-resolutioncomponent synthesizing section 404 and a synthesis ratio determiningsection 405.

Receiving a command from the overall control section 306, the compressedvideo data separation section 402 separates and expands themulti-pixel-resolution compressed video data DMC to thereby obtain thelow-resolution video data LF(t) and the differential video data DF(t).As in FIG. 22, the low-resolution video data LF(t) is processed by theimage size correction/texture conversion section 302, the texturereplacing section 303 and the texture inverse conversion section 304 tobe converted to a high-resolution image HF(t). Note however that thehigh-resolution image HF(t) is produced based only on the low-resolutionvideo data LF(t), and displaying the high-resolution image HF(t) as itis on the display 307 means not taking advantage of the method in whichthe multi-pixel-resolution recorded video F(t) is recorded.

In order to user the recorded original high-resolution image, thehigh-resolution window producing section 403 obtains the originalmulti-pixel-resolution recorded video F(t) by using the differentialvideo data DF(t) and the low-resolution video data LF(t), and outputsthe high-resolution image HWF(t) in the window region. Then, using thehigh-resolution images HF(t) and HWF(t), the high-resolution componentsynthesizing section 404 performs a weighted synthesis process with asynthesis ratio m (m=0 to 1) determined by the synthesis ratiodetermining section 405, to thereby output high-resolution video dataHHF(t) to be displayed on the display 307. The weighted synthesisprocess is performed for the window region, and the high-resolutionimage HF(t) obtained by texture replacing process is used as it isoutside the window region.

HHF(t)=m×HWF(t)+(1−m)×HF(t)  Exp. 4

The overall control section 306 can change the synthesis ratio m bysending the synthesis ratio determining section 405 a signal thatdetermines the synthesis ratio m. Thus, it is possible to reduce thedifference between the image in the window region where the weightedsynthesis process is performed and the image outside the window regionto a level such that the combined image does not seem awkward.

Fourth Embodiment

It is assumed in the third embodiment that the process of learning theresolution increasing parameter is performed in a camcorder during thevideo recording process. In contrast, in the present embodiment, theprocess of learning the resolution increasing parameter is not performedduring the video recording process, but is performed by a server on anetwork or by a display device. This reduces the processing load on thecamera, and the display section can perform the resolution increasingprocess with a high degree of freedom while learning the resolutionincreasing parameter based on the recorded video data.

FIG. 26 is a block diagram showing an exemplary configuration of animage processing device of the present embodiment, wherein the presentinvention is applied to a camcorder. In FIG. 26, like elements to thoseof FIG. 23 are denoted by like reference numerals and will not befurther described below. As compared with the configuration of FIG. 23,an image processing device 51 of FIG. 26 is not provided with a sectionfor learning the resolution increasing parameter, whereby the outputfrom the multi-pixel-resolution image pick-up section 102 is simplyrecorded as the multi-pixel-resolution compressed video data DMC. Thevideo data DMC may be recorded on a medium such as a hard disk, or sentout to a video processing server or an image display section to bedescribed later via a network 501.

FIG. 27 is a block diagram showing the configuration of the videoprocessing server of the present embodiment. In FIG. 27, like elementsto those of FIGS. 23 and 25 are denoted by like reference numerals andwill not be further described below. A video processing server 52 ofFIG. 27 produces the resolution increasing parameter DP from themulti-pixel-resolution compressed video data DMC. The compressed videodata separation section 402 and the high-resolution window producingsection 403 together form a video separation section 410.

As shown in FIG. 27, the compressed video data separation section 402separates and expands the multi-pixel-resolution compressed video dataDMC to thereby obtain the low-resolution video data LF(t) and thedifferential video data DF(t). The high-resolution window producingsection 403 obtains the original multi-pixel-resolution recorded videodata F(t) from the low-resolution video data LF(t) and the differentialvideo data DF(t), and outputs the high-resolution image HWF(t) in thewindow region. The resolution increasing parameter learning section 120obtains the resolution increasing parameter P(t) from thehigh-resolution image HWF(t) and the low-resolution video data LF(t).The resolution increasing parameter recording section 109 stores theresolution increasing parameter P(t) as the resolution increasingparameter DP, or sends it out to the network 501.

In the present embodiment, the high-resolution image is displayed by aconfiguration as shown in FIG. 25 as in the third embodiment, and theprocess will therefore not be further described below.

It is assumed in the present embodiment that the function of learningthe resolution increasing parameter is realized by the video processingserver. Therefore, the function of separating and expanding video dataneeds to be provided also in the video processing server separately fromthe display section. Alternatively, this may be realized by aconfiguration obtained by combining together the video processing serverof FIG. 27 and the image display section of FIG. 25. Then, the functionof separating and expanding video data can be shared by the process oflearning the resolution increasing parameter and the process ofdisplaying the high-resolution image.

Fifth Embodiment

FIG. 28 is a block diagram showing an exemplary configuration of animage processing device according to a fifth embodiment of the presentinvention, wherein the present invention is applied to a camcorder. Theconfiguration of FIG. 28 is an improvement to that of FIG. 8 accordingto the second embodiment. In FIG. 28, like elements to those of FIG. 8are denoted by like reference numerals and will not be further describedbelow.

In the present embodiment, the position of the window region in which ahigh-resolution image is obtained is changed so as to follow the motionof the object instead of simply shifting the window region from oneposition to another over time. Thus, the process can learn theresolution increasing parameter in a concentrated manner in the regionthat does not move out of the position of the object being the focus ofattention, whereby it is possible to improve the precision of theresolution increasing process. The object candidate region can bedetected by analyzing the frequency components of the image. Forexample, the process can define, as an object candidate region, a regionthat contains the most spatial high-frequency components.

As compared with that of FIG. 8, the configuration of FIG. 28additionally includes a previous frame low-resolution frame memory 601,and a window position updating section 160 for detecting the motion ofthe object candidate region in the low-resolution video data and movingthe position of the window region according to the detected motion ofthe object candidate region. The window position updating section 160includes an object candidate detection section 602, an object motiondetection section 603 and a window setting section 604.

The object candidate detection section 602 performs an image filteringprocess for saving high-frequency components on the low-resolution imagestored in the previous frame low-resolution frame memory 601, anddetects a region having a high frequency power as the object candidateregion. Using the low-resolution image at current time t stored in thelow-resolution frame memory 103 and the low-resolution image at time t−1stored in the previous frame low-resolution frame memory 601, the objectmotion detection section 603 performs a block matching process for theobject candidate region detected by an object candidate detectionsection 604 to thereby detect a motion vector. The window settingsection 604 sets a window region based on the motion vector detected bythe object motion detection section 603.

FIGS. 29A to 29D show the process of detecting an object candidateregion. The input image of FIG. 29A is subjected to an edge detectionprocess and a mosaic process to obtain an image of FIG. 29B in which thedegree of frequency power concentration is represented by the shade.Each region of the image of FIG. 29B that includes a position with thedarkest shade is selected as a block, and the process selects an objectcandidate region as shown in FIG. 29C taking into consideration how theblocks are connected to one another. In the illustrated example, theprocess sets the window region as shown in FIG. 29D so that the objectcandidate region is included. While the window region is of an 8×2rectangular shape in the illustrated example, the window region may beof any appropriate shape depending on the configuration of themulti-pixel-resolution image pick-up device.

For the obtained object candidate region, the process calculates themotion vector by using a block matching method between consecutiveframes. Then, based on the obtained motion vector, the process shiftsthe position of the window region to set the new position thereof forthe next point in time t+1. Thus, the process moves the window regionwhere the high-resolution image is obtained so as to follow the motionof the object being the focus of attention.

FIG. 30 shows an example of how to set a window region in the presentembodiment. In FIG. 30, the shape of a window region 2903 is ahorizontal rectangular shape as in FIG. 3. An object 2901 moves aroundin the screen over time, and the process detects in each frame a motionvector 2902 representing the motion of the object 2901. The windowregion 2903 is moved over time according to the motion of the object inthe vertical (V) direction. The position of the window region 2903 doesnot have to follow the order of scan as in FIG. 3, or may not becontinuous between adjacent frames. With the configuration of themulti-pixel-resolution image pick-up section of FIG. 12, the windowregion can be moved in any manner in the vertical (V) direction bychanging the output pulse from the progressive scanning shift register206. With respect to the horizontal (H) direction, the window region maybe made into any shape by discarding unnecessary pixels.

Sixth Embodiment

FIG. 31 is a block diagram showing an exemplary configuration of animage processing device according to a sixth embodiment of the presentinvention, wherein the present invention is applied to a camcorder. Theconfiguration of FIG. 31 is an improvement to that of FIG. 8 accordingto the second embodiment. In FIG. 31, like elements to those of FIG. 8are denoted by like reference numerals and will not be further describedbelow.

In the present embodiment, the low-resolution video data and theresolution increasing parameter are obtained separately for the diffusereflection component and for the specular reflection component.Specifically, the low-resolution video recording section 104 separatelyrecords diffuse reflection component low-resolution video data DLF1 andspecular reflection component low-resolution video data DLF2, and theresolution increasing parameter recording section 109 separately recordsa diffuse reflection component resolution increasing parameter DP1 and aspecular reflection component resolution increasing parameter DP2.Moreover, a multi-pixel-resolution image pick-up section 701 with apolarizing filter is provided instead of the multi-pixel-resolutionimage pick-up section 102 of FIG. 8.

Specifically, the present embodiment uses four image pick-up devices,instead of an ordinary three-chip configuration for color recording, toseparately record the specular reflection component of the object andthe diffuse reflection component thereof and to separately subject thecomponents to the resolution increasing process, to finally combine theimages together. The physical reflection optical characteristics of theobject can be obtained by splitting the specular reflection componentand the diffuse reflection component from the surface luminance. As aresult, it is possible to separately obtain the surface roughness of theobject and the reflectivity inherent to the material such as the albedoof the surface. Therefore, it is very effective in improving the textureof the surface of the object. In view of this, it is expected that thetexture of the surface of the object can be further improved byperforming the resolution increasing process separately for the specularreflection component and for the diffuse reflection component.

While there are various methods for separating the specular reflectioncomponent and the diffuse reflection component from each other, thepresent embodiment employs a method in which a polarizing plate isprovided in front of a camera lens and the components are separated fromeach other from only two images that are obtained with the polarizationaxis being shifted from each other, wherein there are no speciallighting conditions. The method is described in detailed in, forexample, Shinji Umeyama “Separation of diffuse and specular componentsof surface reflection—using multiple observations through a polarizerand probabilistic independence property”, Meeting on Image Recognitionand Understanding (MIRU2002) (pages 1-469 to 476). Therefore, thedetails of the process will not be discussed below, but only theconfiguration of the image pick-up section will be described.

FIG. 32 shows a configuration of the multi-pixel-resolution imagepick-up section 701 with a polarizing filter. As shown in FIG. 32, themulti-pixel-resolution image pick-up section 701 with a polarizingfilter includes a four-way beam splitting prism 711, a redmulti-pixel-resolution image pick-up device 712, a bluemulti-pixel-resolution image pick-up device 713, a first polarizingplate 714, a first green multi-pixel-resolution image pick-up device715, a second polarizing plate 716, a second greenmulti-pixel-resolution image pick-up device 717, a specular-diffusereflection component separation section 718, and an illuminating lightestimation section 719. With this configuration, amulti-pixel-resolution image pick-up device of a wide dynamic range ispreferably used in order to obtain a specular reflection component of avery high contrast.

A multilayer film interference filter, or the like, is sandwiched at theinterface of the four-way beam splitting prism 711. The incident lightis split by a first interface reflection into G (green) light and M(magenta) light, and the M light is split by a second reflection into R(red) light and B (blue) light, which are guided to the redmulti-pixel-resolution image pick-up device 712 and the bluemulti-pixel-resolution image pick-up device 713, respectively. The Glight is split by the second reflection into g1 light, which passesthrough the first polarizing plate 714 and enters the first greenmulti-pixel-resolution image pick-up device 715, and g2 light, whichpasses through the second polarizing plate 716 and enters the secondgreen multi-pixel-resolution image pick-up device 717. While each of thefirst polarizing plate 714 and the second polarizing plate 716 passesonly the polarization primary axis component, the polarization axesthereof are shifted from each other, whereby different images areobtained by the first green multi-pixel-resolution image pick-up device715 and the second green multi-pixel-resolution image pick-up device717. The g1 light and the g2 light, which have passed through thepolarizing plate, are different light from the original G light.

The g1 light and the g2 light are converted by the specular-diffusereflection component separation section 718 into the specular componentGs and the diffuse component Gd of the original G light. Gs and Gd arein the following relationship.

G=Gs+Gd  Exp. 5

The illuminating light estimation section 719 obtains informationregarding the light illuminating the object, by using an AWB (auto whitebalance) function, or the like, which is provided in ordinary cameras.With an object with which a dichromatic reflection model holds, thespecular reflection component Gs is nothing but the G component of theilluminating light. Therefore, where the illuminating light estimationsection 719 estimates the color component ratio (R:G:B) of theillumination (RIL:1:BIL) with G being 1, the specular reflectioncomponents Rs and Bs of red R and blue B can be estimated as follows asthe red R component and the blue B component of the illuminating light.

Rs=RIL·Gs

Bs=BIL·Gs  Exp. 6

Using these, the R light and the B light can also be split into thespecular reflection component and the diffuse reflection component asfollows.

R=Rs+Rd

B=Bs+Bd  Exp. 7

As described above, it is possible to estimate the diffuse reflectioncomponent and the specular reflection component for all of R, G and B.Therefore, these reflection components can be converted by an ordinarymatrix operation into a diffuse reflection component luminance-colordifference space (Yd,Crd,Cbd) and a specular reflection componentluminance-color difference space (Ys,Crs,Cbs).

Therefore, with the configuration of FIG. 31, there are obtained twotypes of video data, i.e., the diffuse reflection componentlow-resolution video data DLF1 and the specular reflection componentlow-resolution video data DLF2, and two types of resolution increasingparameters, i.e., the diffuse reflection component resolution increasingparameter DP1 and the specular reflection component resolutionincreasing parameter DP2.

Normally, when a Y signal is split into a diffuse reflection componentYd and a specular reflection component Ys, multi-pixel-resolutionrecording needs to be performed while splitting each of the RGB primarycolor signals into a specular reflection component and a diffusereflection component, which requires six image pick-up devices. However,the present embodiment, which uses the illuminating light estimationsection 719, can be realized with four image pick-up devices.

FIG. 33 shows a configuration of an image display section of the presentembodiment. The configuration of FIG. 33 has a function of displaying,with a high resolution, video data recorded by an image processingdevice such as a camcorder of FIG. 31. For example, the image displaysection is implemented as a display section of the camcorder or providedin a portable display device or a household large-screen TV set. Thesecond embodiment differs from the configuration of FIG. 22 in that theresolution increasing process is realized separately for the diffusereflection component and the specular reflection component, after whicha diffuse reflection component-specular reflection componentsynthesizing section 702 synthesizes the specular reflection componentand the diffuse reflection component in terms of the luminance and thecolor difference level. The synthesis process may be performed either inthe luminance-color difference YCrCb space obtained by inverseconversion from texture or in the primary color RGB space. Video inputsections 301 a and 301 b operate as does the video input section 301 ofFIG. 22. Moreover, image size correction/texture conversion sections 302a and 302 b, texture replacing sections 303 a and 303 b, texture inverseconversion sections 304 a and 304 b and resolution increasing parameterinput sections 305 a and 305 b operate as do the image sizecorrection/texture conversion section 302, the texture replacing section303, the texture inverse conversion section 304 and the resolutionincreasing parameter input section 305 of FIG. 22, respectively.

It is considered in the present embodiment that the diffuse reflectioncomponent and the specular reflection component are separate andindependent signals, and each of them is subjected to the resolutionincreasing process separately. Therefore, the present embodiment can becombined with the second to fifth embodiments described above. Moreover,the diffuse reflection component low-resolution video data DLF1 and thespecular reflection component low-resolution video data DLF2 may becombined together and stored as a single image format.

In the present embodiment, the resolution increasing process for thediffuse reflection component and that for the specular reflectioncomponent are performed by the same method. However, they do not need tobe performed by the same method, but may alternatively be performed bydifferent methods. For example, an interpolating enlargement methodknown in the art may be used only for the diffuse reflection component.

Moreover, it may not be necessary to subject both of the diffusereflection component and the specular reflection component to theresolution increasing process. Where only one of the diffuse reflectioncomponent and the specular reflection component is subjected to theresolution increasing process, the selection between the two componentscan be made based on the characteristics of the object, the light sourceenvironments used for recording, the purpose of the resolutionincreasing process, etc. For example, the specular reflection componentmay be selected in order to emphasize the irregularities of the surfaceof a craft product or an object subjected to a special surfacetreatment, and the diffuse reflection component may be selected in orderto faithfully reproduce text printed on the surface of the object.

FIG. 34 schematically shows a resolution increasing process of thepresent embodiment. Assume a spherical object with surfaceirregularities and with text “ABC” printed thereon. Referring to FIG.34, an input low-resolution image 3301 has a highlight 3307 over thetext due to specular reflection of light. This can be split into aspecular reflection component 3302 and a diffuse reflection component3303, wherein the text portion being a result of diffuse reflectiondisappears in the specular reflection component 3302, whereas thehighlight portion and the surface irregularity disappear in the diffusereflection component 3303. Then, as the specular reflection component3302 is subjected to the resolution increasing process, there isproduced an image 3304 in which the surface irregularity and thespecular reflection component have been resolution-increased with thetext portion remaining disappeared. As the diffuse reflection component3303 is subjected to the resolution increasing process, there isproduced an image 3305 in which only the text is resolution-increasedwhile the specular reflection and the highlight disappear. From theimages 3304 and 3305, it is possible to produce as an output image ahigh-resolution image 3306. The image 3306 carries a larger amount ofinformation and is visually a better result as compared with a casewhere an original image is subjected to the resolution increasingprocess as it is without splitting the original image into a diffusereflection component and a specular reflection component.

The present invention realizes, with a small amount of informationprocessing, a high-precision resolution conversion in which objectcharacteristics are reflected in the resultant image, and is thereforeeffective in producing digital high-resolution video data with richtexture in various applications where the amount of visual informationvalued highly, for example. Moreover, the present invention is alsoeffective in displaying an image on a low-resolution display system suchas a mobile telephone, for example.

1. An image recording method, comprising a step of obtaining andrecording multi-pixel-resolution video data by taking an image with ahigh resolution in a window region being a portion of an entire imagewhile taking an image with a low resolution outside the window region,wherein a position of the window region is changed every frame in thestep.
 2. An image processing device, comprising: amulti-pixel-resolution image pick-up section for obtainingmulti-pixel-resolution video data by taking an image with a highresolution in a window region being a portion of an entire image andwhose position changes every frame while taking an image with a lowresolution outside the window region; a downward resolution conversionsection for performing an image filtering process on the high-resolutionimage in the window region to decrease a resolution of thehigh-resolution image; a low-resolution video recording section forrecording low-resolution video data obtained from an output of thedownward resolution conversion section and the multi-pixel-resolutionvideo data; a resolution increasing parameter learning section forlearning a resolution increasing parameter by using the high-resolutionimage in the window region and the output of the downward resolutionconversion section; and a resolution increasing parameter recordingsection for recording the resolution increasing parameter.
 3. The imageprocessing device of claim 2, wherein the multi-pixel-resolution imagepick-up section obtains the multi-pixel-resolution video data while themulti-pixel-resolution video data is split into a luminance componentand a color difference component.
 4. The image processing device ofclaim 2, further comprising a record button, wherein when the recordbutton is pressed, the multi-pixel-resolution image pick-up sectionrecords an image, and the low-resolution video data and the resolutionincreasing parameter are recorded.
 5. The image processing device ofclaim 2, further comprising a window position updating section fordetecting a motion of an object candidate region in the low-resolutionvideo data and moving a position of the window region according to thedetected motion of the object candidate region.
 6. The image processingdevice of claim 5, wherein the window position updating section detectsthe object candidate region based on a frequency analysis of an image.7. The image processing device of claim 2, wherein themulti-pixel-resolution image pick-up section obtains themulti-pixel-resolution video data while the multi-pixel-resolution videodata is split into a diffuse reflection component and a specularreflection component.
 8. The image processing device of claim 7, whereinthe multi-pixel-resolution image pick-up section splits a taken colorimage into a plurality of primary color components and splits one of theprimary color components into a specular reflection component and adiffuse reflection component, and wherein a diffuse reflection componentand a specular reflection component of the color image are obtained byusing the obtained specular reflection component and diffuse reflectioncomponent.
 9. The image processing device of claim 7, wherein themulti-pixel-resolution image pick-up section estimates illuminatinglight used for recording, and the specular reflection component isobtained by using the estimated illuminating light.
 10. An imageprocessing device for performing a resolution increasing process byusing the low-resolution video data and the resolution increasingparameter recorded by the image processing device of claim 2, the imageprocessing device comprising: a video input section for reading thelow-resolution video data; a resolution increasing parameter inputsection for reading the resolution increasing parameter; and aresolution increasing section for increasing a resolution of thelow-resolution video data read by the video input section by using theresolution increasing parameter read by the resolution increasingparameter input section.
 11. An image processing device, comprising: amulti-pixel-resolution image pick-up section for obtainingmulti-pixel-resolution video data by taking an image with a highresolution in a window region being a portion of an entire image andwhose position changes every frame while taking an image with a lowresolution outside the window region; a downward resolution conversionsection for performing an image filtering process on the high-resolutionimage in the window region to decrease a resolution of thehigh-resolution image; a multi-pixel-resolution video recording sectionfor recording the multi-pixel-resolution video data; a resolutionincreasing parameter learning section for learning a resolutionincreasing parameter by using the high-resolution image in the windowregion and the output of the downward resolution conversion section; anda resolution increasing parameter recording section for recording theresolution increasing parameter.
 12. The image processing device ofclaim 11, wherein the multi-pixel-resolution video recording sectionrecords the multi-pixel-resolution video data while themulti-pixel-resolution video data is split into low-resolution videodata and differential video data representing a difference between themulti-pixel-resolution video data and the low-resolution video data. 13.An image processing device, comprising: a multi-pixel-resolution imagepick-up section for obtaining multi-pixel-resolution video data bytaking an image with a high resolution in a window region being aportion of an entire image and whose position changes every frame whiletaking an image with a low resolution outside the window region; and amulti-pixel-resolution video recording section for recording themulti-pixel-resolution video data while the multi-pixel-resolution videodata is split into low-resolution video data and differential video datarepresenting a difference between the multi-pixel-resolution video dataand the low-resolution video data.
 14. An image processing device forobtaining a resolution increasing parameter by usingmulti-pixel-resolution video data recorded by another image processingdevice comprising a) a multi-pixel-resolution image pick-up section forobtaining multi-pixel-resolution video data by taking an image with ahigh resolution in a window region being a portion of an entire imageand whose position changes every frame while taking an image with a lowresolution outside the window region and b) a multi-pixel-resolutionvideo recording section for recording the multi-pixel-resolution videodata while the multi-pixel-resolution video data is split intolow-resolution video data and differential video data representing adifference between the multi-pixel-resolution video data and thelow-resolution video data, the image processing device comprising: avideo separation section for reading the multi-pixel-resolution videodata and separating the low-resolution video data therefrom and forobtaining the high-resolution image in the window region by using thedifferential video data; a resolution increasing parameter learningsection for learning a resolution increasing parameter by using thehigh-resolution image in the window region and the low-resolution videodata; and a resolution increasing parameter recording section forrecording the resolution increasing parameter.